🛠 Improved Twitter Ingestion

Target Release: 08/30/24

🚀 Key Features:

  • Sampling for Large Requests: Automatically manage the sample rate to maintain a daily limit of 20k tweets per query.
  • Flexible Workspace Queries: Create workspaces with basic or advanced queries using special operators.
  • Automatic Backfill: Automatically backfill the last 30 days of tweets with configurable daily limits.

🔍 Why It Matters:

Our clients, including commercial clients and government agencies, expect comprehensive and timely coverage of relevant tweets. The Twitter Ingestion feature ensures that workspaces remain within rate limits while optimizing tweet ingestion, avoiding the risk of missing critical narratives.

Technical Integration:

  1. Workspace Management: Create, edit, and delete workspaces that include Twitter (X) with seamless integration into our data pipeline.
  2. Rate Limits Adherence: Automatically adjusts query sampling to adhere to Twitter API rate limits.

Risks:

  • Complex Queries: May fail if not translated properly to Twitter syntax.
  • Quota Management: Overlapping queries could lead to excessive quota usage.